Skip to content

HDDS-5060. [SCM HA Security] Make InterSCM grpc channel secure.#2187

Merged
bshashikant merged 3 commits intoapache:masterfrom
bharatviswa504:HDDS-5060
Apr 28, 2021
Merged

HDDS-5060. [SCM HA Security] Make InterSCM grpc channel secure.#2187
bshashikant merged 3 commits intoapache:masterfrom
bharatviswa504:HDDS-5060

Conversation

@bharatviswa504
Copy link
Contributor

@bharatviswa504 bharatviswa504 commented Apr 27, 2021

What changes were proposed in this pull request?

Make InterSCM grpc channel secure.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-5060

How was this patch tested?

Tested manually on docker ozone-secure-ha

Started Scm3 later, after some transactions are performed.

Config changes for testing:
OZONE-SITE.XML_ozone.scm.ha.ratis.log.purge.enabled=true
OZONE-SITE.XML_ozone.scm.ha.ratis.log.purge.gap=10
OZONE-SITE.XML_ozone.scm.ha.ratis.snapshot.threshold=5

scm3.org_1   | 2021-04-27 07:27:57,895 [grpc-default-executor-3] INFO server.GrpcServerProtocolService: 01c9e353-3886-4c6a-8c9b-f7278a473f30: Completed INSTALL_SNAPSHOT, lastRequest: 5249309c-73b8-4f8f-9b08-5cd73257f1df->01c9e353-3886-4c6a-8c9b-f7278a473f30#0-t2,notify:(t:2, i:43)
scm3.org_1   | 2021-04-27 07:27:57,929 [grpc-default-executor-0] INFO server.RaftServer$Division: 01c9e353-3886-4c6a-8c9b-f7278a473f30@group-EFC7205245FB: Failed appendEntries as snapshot ((t:2, i:43)) installation is in progress
scm3.org_1   | 2021-04-27 07:27:57,930 [grpc-default-executor-0] INFO server.RaftServer$Division: 01c9e353-3886-4c6a-8c9b-f7278a473f30@group-EFC7205245FB: inconsistency entries. Reply:5249309c-73b8-4f8f-9b08-5cd73257f1df<-01c9e353-3886-4c6a-8c9b-f7278a473f30#47:FAIL-t2,INCONSISTENCY,nextIndex=1,followerCommit=-1
scm3.org_1   | 2021-04-27 07:27:57,931 [grpc-default-executor-3] INFO server.RaftServer$Division: 01c9e353-3886-4c6a-8c9b-f7278a473f30@group-EFC7205245FB: receive installSnapshot: 5249309c-73b8-4f8f-9b08-5cd73257f1df->01c9e353-3886-4c6a-8c9b-f7278a473f30#0-t2,notify:(t:2, i:43)
scm3.org_1   | 2021-04-27 07:27:57,931 [grpc-default-executor-3] INFO server.RaftServer$Division: 01c9e353-3886-4c6a-8c9b-f7278a473f30@group-EFC7205245FB: Snapshot Installation by StateMachine is in progress.
scm3.org_1   | 2021-04-27 07:27:57,931 [pool-13-thread-1] INFO ha.SCMHAManagerImpl: Reloaded SCM state with Term: 2 and Index: 48

bshashikant
bshashikant approved these changes Apr 27, 2021
View reviewed changes
Copy link
Contributor

@bshashikant bshashikant left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me. @xiaoyuyao , can you please check this one?

xiaoyuyao
xiaoyuyao approved these changes Apr 27, 2021
View reviewed changes
Copy link
Contributor

@xiaoyuyao xiaoyuyao left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, +1.

vivekratnavel
vivekratnavel approved these changes Apr 27, 2021
View reviewed changes
Copy link
Contributor

@vivekratnavel vivekratnavel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 LGTM

@bshashikant bshashikant merged commit 6f9c3f2 into apache:master Apr 28, 2021
@bshashikant
Copy link
Contributor

bshashikant commented Apr 28, 2021

Thanks @bharatviswa504 for the fix and @xiaoyuyao / @vivekratnavel for the review. Committed this.

errose28 added a commit to errose28/ozone that referenced this pull request May 4, 2021
@errose28
Merge remote-tracking branch 'upstream/master' into HDDS-3698-nonroll…
253cc55 
…ing-upgrade-master-merge2

* upstream/master: (56 commits)
  HDDS-2212. Genconf tool should generate config files for secure clust… (apache#1788)
  HDDS-5166. Remove duplicate assignment of OZONE_OPTS for freon and sh (apache#2195)
  Revert "HDDS-5144. Create github check to alert when dependency tree is changed (apache#2177)"
  HDDS-4983. Display key offset for each block in command key info (apache#2051)
  HDDS-5144. Create github check to alert when dependency tree is changed (apache#2177)
  HDDS-4585. Support bucket acl operation in S3g (apache#1701)
  HDDS-5153. Decommissioning a dead node should complete immediately (apache#2190)
  HDDS-5147. Intermittent test failure in TestContainerDeletionChoosingPolicy#testRandomChoosingPolicy (apache#2188)
  HDDS-5152. Fix Suggested leader in Client. (apache#2189)
  HDDS-5148. Bump ratis version to 2.1.0-ff8aa66-SNAPSHOT (apache#2184)
  HDDS-4515. Datanodes should be able to persist and load CRL (apache#2181)
  HDDS-5060. [SCM HA Security] Make InterSCM grpc channel secure. (apache#2187)
  HDDS-5051. Ensure failover to suggested leader if any for NotLeaderException. (apache#2141)
  HDDS-5127. Fix getServiceList when SCM HA is enabled (apache#2173)
  HDDS-4889. Add simple CI check for docs (apache#2156)
  HDDS-5131. Use timeout in github actions (apache#2176)
  HDDS-5103. Fix Install Snapshot Mechanism in SCMStateMachine. (apache#2155)
  HDDS-5124. Use OzoneConsts.OZONE_TIME_ZONE instead of "GMT" (apache#2166)
  HDDS-5047. Refactor Pipeline to use ReplicationConfig instead of factor/type (apache#2096)
  HDDS-5083. Bump version of common-compress (apache#2139)
  ...

Conflicts:
	hadoop-hdds/common/pom.xml
	hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConfigKeys.java
	hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConsts.java
	hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/HddsDatanodeService.java
	hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/SCMNodeManager.java
	hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/PipelineManager.java
	hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMStorageConfig.java
	hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/StorageContainerManager.java
	hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/node/TestSCMNodeManager.java
	hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/MiniOzoneClusterImpl.java
	hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OMStorage.java
	hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
	hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/scm/ReconStorageContainerManagerFacade.java
errose28 added a commit to errose28/ozone that referenced this pull request May 13, 2021
@errose28
Merge branch 'HDDS-3698-nonrolling-upgrade' into HDDS-5138-admin-chec…
080019b 
…k-in-auth

* HDDS-3698-nonrolling-upgrade: (57 commits)
  Fix compilation errors afte merge Update javassist in recon pom Fix changes introduced in merge that failed TestSCMNodeManager upgrade tests Fix checkstyle Fix intermittent test failure TestSCMNodeManager#testSetNodeOpStateAndCommandFired after merge Skip scm init default layout version in TestOzoneConfigurationFields
  HDDS-2212. Genconf tool should generate config files for secure clust… (apache#1788)
  HDDS-5166. Remove duplicate assignment of OZONE_OPTS for freon and sh (apache#2195)
  Revert "HDDS-5144. Create github check to alert when dependency tree is changed (apache#2177)"
  HDDS-4983. Display key offset for each block in command key info (apache#2051)
  HDDS-5144. Create github check to alert when dependency tree is changed (apache#2177)
  HDDS-4585. Support bucket acl operation in S3g (apache#1701)
  HDDS-5153. Decommissioning a dead node should complete immediately (apache#2190)
  HDDS-5147. Intermittent test failure in TestContainerDeletionChoosingPolicy#testRandomChoosingPolicy (apache#2188)
  HDDS-5152. Fix Suggested leader in Client. (apache#2189)
  HDDS-5148. Bump ratis version to 2.1.0-ff8aa66-SNAPSHOT (apache#2184)
  HDDS-4515. Datanodes should be able to persist and load CRL (apache#2181)
  HDDS-5060. [SCM HA Security] Make InterSCM grpc channel secure. (apache#2187)
  HDDS-5051. Ensure failover to suggested leader if any for NotLeaderException. (apache#2141)
  HDDS-5127. Fix getServiceList when SCM HA is enabled (apache#2173)
  HDDS-4889. Add simple CI check for docs (apache#2156)
  HDDS-5131. Use timeout in github actions (apache#2176)
  HDDS-5103. Fix Install Snapshot Mechanism in SCMStateMachine. (apache#2155)
  HDDS-5124. Use OzoneConsts.OZONE_TIME_ZONE instead of "GMT" (apache#2166)
  HDDS-5047. Refactor Pipeline to use ReplicationConfig instead of factor/type (apache#2096)
  ...
bharatviswa504 added a commit to bharatviswa504/hadoop-ozone that referenced this pull request Jul 25, 2021
@bharatviswa504 @bshashikant
HDDS-5060. [SCM HA Security] Make InterSCM grpc channel secure. (apac…
60328d2 
…he#2187)

(cherry picked from commit 6f9c3f2)
Change-Id: I9c17c129cbe3c7fdb20f5c8121740bc5b67ff6f3
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@vivekratnavel vivekratnavel vivekratnavel approved these changes

@xiaoyuyao xiaoyuyao xiaoyuyao approved these changes

@bshashikant bshashikant bshashikant approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@bharatviswa504 @bshashikant @vivekratnavel @xiaoyuyao

Comments